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ABSTRACT 


This  study  reports  on  the  feasibility  of  automatic  processing  of  occupational 
data  recorded  on  IBM  mark-sense  cards  as  an  alternative  to  key  punching  the  data 
wrinen  in  inventory  booklets.  3  achninistrative  procedures  were  used  in  collecting 
information  from  367  Air  Police  (77XXX)  incumbents.  Comparisons  of  accuracy, 
processing  time,  and  costs  were  made  atross  data-processing  methods,  skill  level, 
and  administrative  procedure.  Vith  carefully  designed  inventory  anu  card  formats, 
visual  scanning,  machine  editing,  and  top  maintenance  of  the  IBM  reproducer,  the 
mark-sense  technique  was  found  feasible,  but  more  expensive  than  the  key-punch 
method.  The  2  administrative  techniques,  in  which  incumbents  marked  whether 
they  performed  each  task  in  the  inventory  before  adding  unlisted  tasks,  elicited 
twice  as  many  write-in  statements  as  the  third  technique,  rrhere  incumbents  were 
merely  to  read  the  listed  statements  before  adding  write-ins.  When  the  incumbents 
rated  tasks  for  bodi  amount  of  time  spent  and  training  required,  die  correlation  be¬ 
tween  the  ratings  was  lowest  when  the  first  ratings  were  not  visible  during  the 
second  rating. 
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USING  MARK-SENSE  CARDS  FOR  COLLECTING  OCCUPATIONAL  INFORMATION 


I.  PROBLEM 

The  job  inventory  method  of  collecting,  organizing,  and  reporting  information  about  work 
performed  by  incumbents  is  presented  in  Air  Force  Manual  35-2-  The  present  investigation  is 
one  of  a  series  of  studies  designed  to  further  the  development  of  procedures  for  obtaining 
quantifiable  data  from  )ob  inventories  and  punching  these  data  into  electronic  data  processing 
machine  (EDPM)  cards. 

During  the  early  stages,  1959-1961,  of  the  development  of  the  Air  Force  occupational  analy¬ 
sis  method,  it  appeared  that  die  process  of  key  punching  the  inventory  liata  by  hand  might  become 
a  major  source  of  delay.  The  large  amount  of  key  punching  required  at  that  time  prompted  an 
investigation  of  automatic  mark-sense  punching  technique  as  a  possible  alteniative.  In  addition, 
certain  variations  of  inventory  administration  were  in  need  of  experimental  trial. 

This  study  accordingly  had  two  purposes.  One  wac  to  compare  the  mark-sense  method 
with  the  currently  used  ke>’ -punch  method  in  accuracy  and  economy  of  punching  occupational 
survey  data  into  cards.  The  other  was  to  compare  three  procedures  for  rating  work  tasks. 

In  late  1959  (McCormick  St  Ammerman,  I960)  a  job  inventory  booklet  format  was  developed 
with  photograph  mounting  corners  for  holding  mark-sense  task-rating  cards.  The  cost  of  these 
reustiable  booklets,  however,  was  excessive;  and  the  time  requued  for  loading  and  unloading 
the  cards  was  considered  prohibitive.  An  attempt  was  made  to  eliminate  these  objections  in 
the  present  study. 


2.  PROCEDURES 

Pretests 

In  early  1959,  a  standard  mark-sense  card  for  recording  background  information  (BI)  bad 
been  developed  to  be  used  with  mark-sense  test-item  cards  in  the  administration  of  experimental 
tests.  Along  the  bonon  of  this  card  were  18  boxes  for  the  examinee  to  write  his  name  according 
to  specifications  for  EDPM  use.  A  column  of  ovals  over  each  box  was  provided  for  mark-sensing. 
In  i960,  this  BI  card  was  tried  out  on  a  flight  of  basic  airmen  to  determine  its  suitability  for 
occupational  surveys.  Tfie  card  was  found,  however,  to  take  a  great  deal  of  time  and  to  result 
in  many  errors,  particularly  in  writing  and  mark -sensing  the  name. 

Subsequently  the  mark-sense  BI  card  and  a  mark-sense  card  desif^ued  for  recording  task 
ratings  were  tried  out.  Changes  in  administrative  procedures  for  the  BI  card  included  simplified 
steps  for  arritine  and  mark-sensing  the  name,  and  the  provision  on  a  blackboard  of  samples  of 
good  and  poor  mark-sensing,  plus  a  list  of  coded  job  assignment  titles.  The  job  inventory  used 
for  this  tryout  was  one  for  Air  Police  (AFSC  77XXX)  consisting  of  146  usk  statements  grouped 
under  13  duty  headings.  This  inventory  was  assembled  primarily  from  materials  previously 
used  (Wiley,  1959)-  1°  order  to  facilitate  ali^ment,  the  task  statements  were  typed  so  that 
the  spacing  matched  the  mark-sense  card  columns  and  were  numbered  in  both  margins.  The 
pages  of  the  booklet  were  printed  on  one  side  only,  and  each  duty  began  on  a  new  page.  Write- 
in  tasks,  those  performed  by  the  incumbent  but  not  listed,  were  to  be  added  on  the  pages  of  the 
booklet.  When  the  inventory  was  administered  to  17  members  of  an  Air  Force  Police  flight,  it 
was  found  that  the  BI  card,  including  the  name,  took  too  long  to  complete.  In  addition,  the  use 
of  bodi  sides  of  the  task-rating  cards  for  responding  to  tasks  of  die  two  longer  duties,  and  the 
necessity  for  turning  these  cards  back  before  the  next  step  of  die  surv*v  could  be  taken,  inter¬ 
fered  uth  smoodi  administration. 
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la  the  light  of  the  earlier  trials,  the  loTentory  booklet,  bod«  the  BI  and  task-rating  cards, 
and  the  administrative  directions  were  revised. 

Inventory  Booklet.  Duties  and  tasks  of  the  inventory  used  in  the  pretest  were  retained, 
but  the  following  changes  were  made  in  format.  The  examples  of  mark -sensing  and  the  list  of 
coded  job  titles,  formerly  placed  on  a  blackboard,  were  printed  on  the  lirst  page  of  the  booklet, 
and  an  illuscration  of  the  BI  card  properly  filled  out  was  added  (see  Figure  1).  In  an  effort  to 
reduce  the  visual  demands  of  responding  to  the  task  statements,  a  type  foot  vtas  used  that 
re<iuired  only  one  line  for  each  statement,  and  the  statements  were  typed  by  reverse  composi¬ 
tion  so  that  they  ended  near  the  right-hand  margin  of  the  page,  as  shown  in  Figure  2.  A  maxi¬ 
mum  of  23  task  statements  were  listed  on  each  page. 

MarkSenMe  BI  Card.  The  front  of  the  revised  mark-sense  BI  card  was  printed  with  item 
headings  at  the  top,  and  a  row  of  boxes  for  writing  digits,  as  shown  in  Figure  1.  Provision 
was  made  for  omrk-scnsing  only  one  letter,  the  specialty  shredout  (which  does  not  occur  in 
the  Air  Police  career  field).  For  identification  and  ready  card  handling,  three  items  were  pre¬ 
punched  and  interpreted  (machine  printed):  first  two  digits  (77)  of  die  AFSC,  beginning  date 
of  the  study,  and  case  control  number. 

The  troublesome  writing  and  mark-sensing  of  the  name  was  eliminated  by  designing  the 
back  of  the  BI  card  primarily  for  items  to  be  written  out  but  not  mark-sensed.  In  addition  to 
space  for  the  incumbent’s  name,  the  back  of  the  card  provided  for  writing  the  Air  Force  service 
number,  Air  Force  base,  and  squadron  or  unit;  sod  had  extra  lines  for  other  information.  Items 
such  as  these,  rarely  used  directly  in  analysis,  were  provided  so  that  they  could  be  used  for 
identification,  to  supplement  the  case  control  number,  or  for  whatever  cross-referencing  might 
be  desired.  Eight  additional  mark-sense  columns  were  included  on  the  back  of  the  card  for 
miscellaneous  purposes.  The  extra  lines  and  mark-sense  columns,  not  used  in  she  present 
study,  were  provided  to  make  the  card  suiuble  for  general  occupational  survey  purposes. 

Mark-Sense  Task-Rating  Cards.  As  revised,  the  front  of  each  mark-sense  task-rating 
cord  had  spaces  tot  rating  a  maximum  of  23  tasks;  duties  with  over  23  tasks  had  two  rating 
cards.  One  mark-sense  card  column  was  provided  for  responding  to  each  task.  The  column 
had  two  sets  of  five  ovals  each  for  the  two  ratings,  plus  an  oval  on  the  right  for  designating 
the  task  as  done  or  not  done.  Cards  were  reproduced  in  two  slightly  different  formats:  one 
for  the  separate-step  treatments  (covered  and  uncovered  rating)  with  the  right-hand  oval  to  be 
tuuksd  for  each  task  performed  by  the  incumbent;  and  the  other  tor  the  single  step  treatment 
widh  the  right-hand  oval  to  be  marked  tot  each  task  not  perfcrmed.  (Figure  3  shows  a  card 
for  the  covered  rating  treatment,  designated  by  the  letter  X  iu  the  case  control  number.)  The 
backs  of  the  usk-rating  cards  were  to  be  used  by  the  incumbents  for  adding  write-in  task 
statements. 

Fifteen  usk-rating  cards,  one  for  each  page  of  the  inventory,  were  provided  for  each 
incumbent.  Of  the  13  duties,  A  through  M,  all  except  two  had  23  or  fewer  tasks  and  required 
one  booklet  page  and  one  corresponding  card.  Duties  D  and  F  had  30  tasks  each  and  required 
two  booklet  pages  and  two  cards.  One  additional  card  was  included  for  tasks  of  added  write-in 
duties  and  for  miscellaneous  write-in  tasks.  Thus  a  set  for  each  incumbent  had  17  cards:  a 
BI  card,  13  task-rating  cards,  and  a  card  for  additional  write-in  tasks. 

Sampling.  The  inventory  was  administered  to  373  incumbents  of  the  Air  Police  Career 
Field  (AFSC  77XXX)  in  groups  of  3  to  23,  assigned  to  four  installations  near  San  Antonio, 
Texas.  In  order  to  obtain  equated  subsamples  for  the  diree  treatments,  incumbents  available 


2 


AF$C.  fST  2  DIGITS; 


•  11/09 


e-c6 


MANO 


iJL, 


VR9  IM 
ACT  990 
WlL  ASIIV 


t3Cl=rt3Cl 
23C2I  :2^.2 

:33c34:3>:3 


JDCI: 

±Jx-'2+ 


r«44:^Th4:. 


5:>c5rfc5 


6^6 


6>-‘C- 

7:>c7rj-.7=c7rt:7oc7 
8-v3 


U-  -V 

q^2 


\  ±l1. 


!-8-^6 


V 


c5t 


oT? 

HMOMOi 


r«c« 

LAST  • 
OlCir*  AND 


1^1 
:23c2>r2 
[:3dc33c3 
4dc43c4 

j:63c63c6 
7dc7^7 
8:c8  c8 


-UiL 


ImL 


ill 


+4 


F|3c|dc1 

-23c2oc2 
|-33c3^3 

43«4*c4 

53c5:xi5-|: 

e^cs^ceii 

-g^rg  yQi^  fljcgocq 


fJl 


m 


7mt\ 


Mi 


SURVEY  DATE 


MO*  <M 
A**C 


■  I* 


<*>1  Vl^ 


..lim,,  6..1. 


CONTROL  NO: 


|RRfi*CNt  MaA  to 
MTOIIM  FlAAAto  OaRi 

a**icm  AaaIha 


fe^D  j«o»^o^o 

iDclrfclDcfocl 

23c2i2=Oc^ 
3^34:33c3^c3 
-)r44-43c4r' 

5dc5  isscSJcSi 

6^S-^6^^s4^ 

7ix:7i7Dc7oc7 

B-*©-*!  8^8^8 


OW 


b\^\L/ 


Avm>«c«  wtiKLT  ouTy  Moum 
,a.,.  I  to  I  WMk  «•  ( 

^  jf^r  AfAAjfHtoy  fMiMttAAA 


-0>g^' 0^0-{*0*g;>afCg^  ft 
:  r|3ciil3cl::fclDCt::fct>:|; 

2x2=f2x:2=t2x2:^2>^2- 

3x3::^^:tc3:i=2:}c3j!c3^3: 

:^x4±4zc4x4:>^ 
5^5=f5x5x5x5xix5 
63c0-:i:0D4^cgDcS  ^  6- 

yxjxjxl^jxjxjxj . 
flocSJ.  B^cg^cgxg:'.  8>^8 
:j  3^1>9  ^9---c9|-9-)-9-K9y3-.'  9%  g 


Fig.  1.  BI  Card,  Fraat 
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Fig.  2.  Saaple  of  forraat  for  listing  task  statenents. 
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for  each  survey  session  were  stratified  into  categories:  first  by  Primary  AFSC;  then,  within 
each  AFSC,  by  military  grade  (for  example,  77150-staff  sergeants  (E-5)).  During  the  course 
of  the  survey,  numbers  of  incumbents  listed  as  available  for  a  given  session  were  distributed 
among  the  three  treatments  so  tha.  the  cumulative  totals  of  the  categories  were  kept  as  nearly 
equal  as  possible.  Individuals  to  be  assigned  on  this  basis  to  the  three  treatments  were  se¬ 
lected  by  use  of  a  table  of  random  numbers.  Inequalities  of  treatment  distributions  resulting 
from  absentees  and  extra  incumbents  at  a  survey  session  were  adjusted  in  making  assignments 
for  the  following  sessions.  Although  there  were  125  cases  in  each  of  the  treatments,  eight 
cases  of  the  covered  rating  treatment  had  to  be  discarded  because  of  an  error  in  administration. 
Only  9  of  die  367  remaining  incumbents  were  in  the  Correction  Ladder  (77XX1);  the  other  358 
were  in  the  conventional  Air  Police  Ladder  (77XX0).  Moreover,  the  9  skill  level  had  only  one 
case;  and  the  1  skill  level  only  11;  the  other  355  were  the  7,  5,  and  3  levels. 

The  specialty  (Primary  AFSC)  and  grade  distributions  for  the  three  treatments,  shown  in 
Table  1,  indicate  a  hi^  degree  of  equivalence  as  estimated  by  the  chi-square  test  for  distri¬ 
butions  assumed  to  have  been  randomly  selected  from  the  same  parent  populr.tion.  For  computa¬ 
tion  of  chi  square,  small  AFSC  entries  were  combined  with  larger  entries  to  form  three  groups: 

9  and  7  skill  level,  5  level,  and  3  and  1  level.  Similarly  combinations  were  made  of  the  top 
four  and  the  bottom  two  grade  entries  to  give  five  groups. 


Table  1.  Distribatioa  of  Treatmeats  by  AFSC  aad  by  Grade 


AFSC 

TREATMENT 

GRADE 

TREATMENT 

cov 

UNCOV 

1  STEP 

COV 

UNCOV 

1  STEP 

[77190 

0 

0 

1 

[E-9 

0 

0 

0 

77171 

2 

2 

0 

E-8 

0 

0 

1 

[77170 

10 

10 

11 

E-7 

1 

1 

0 

'77151 

1 

1 

3 

.E-6 

4 

5 

4 

.77150 

83 

86 

88 

E-5 

26 

23 

24 

77131 

0 

0 

0 

E-4 

34 

41 

39 

77130 

18 

23 

17 

E-3 

37 

40 

41 

.77010 

3 

3 

5 

E-2 

15 

14 

15 

li-1 

0 

1 

1 

Total 

117 

125 

125 

Total 

117 

125 

125 

X*  =  .55,  df 

-  4,  P 

=  .97 

X*  *  .93,  df 

=  8,  P  >  .99 

A/ore.— Bracketed  entries  combined  in  computing 


Treatments,  The  covered  rating  treatment  provided  for  five  steps  of  administration: 

(a)  filling  out  and  mark-sensing  the  BI  card;  (b)  reading  die  tasks  and  mark-sensing  the  "Mark 
if  done**  oval  for  each  task  performed;  (c)  adding  write-in  tasks  on  the  backs  of  task-rating 
cards  and  on  die  extta  write-in  card;  (d)  rating  every  task  performed  on  total  time  spent;  and 
(e)  rating  every  task  perfomied  on  training  and  experience  required  to  accomplish  the  task  ade¬ 
quately.  All  ratings  were  to  be  made  on  an  ipsative  basis.  The  second  rating  was  made  with 
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the  rating  card  partly  under  the  inventory  page  so  that  the  first  rating  could  not  be  seen.  Tliis 
treatment  was  designed  to  exclude  opportunity  for  the  second  rating  of  a  ta»k  to  be  influenced 
by  the  first  rating. 

The  uncovered  rating  treatment  consisted  of  the  same  five  steps  as  the  covered  rating 
treatment,  with  one  difference:  the  second  rating  was  mads  without  covering  die  first  rating. 

The  likelihood  of  the  second  rating  being  influenced  by  the  first  was  expected  to  be  greater 
than  in  the  covered  rating  treatment. 

The  single-step  rating  treatment  required  the  incumbent:  to  complete  the  BI  card;  to  read 
the  listed  tasks  and  add  write-in  tasks;  and  for  each  task,  cither  to  mark  the  oval  labeled  “Mark 
if  NOT  done,”  or  to  make  the  two  ratings  before  going  on  to  the  next  task.  The  single  step 
rating  treatment  proved  to  be  the  most  easily  and  quickly  administered.  This  treatment,  however, 
was  expected  to  offer  the  greatest  likelihood  of  the  second  rating  being  influenced  by  the  first. 

Administration.  A  set  of  administrative  directions  was  prepared  for  each  treatment.  Ex¬ 
perienced  test  administrators  conducted  the  survey  sessions.  They  were  assigned  to  the  three 
treatments  in  rotation  to  keep  the  conditions  of  administration  as  nearly  balanced  as  possible. 

Data  Processing.  The  visual  hand-scanning  and  correction  prior  to  machine  processing 
included:  erasing  multiple  responses  to  the  same  factor,  responses  where  only  one  factor  was 
rated,  stray  and  poorly  erased  marks  and  marks  extending  beyond  ovals;  blackening  light  marks; 
and  marking  BI  data  recorded  in  writing  but  not  mark-sensed. 

Machine  processing  was  done  in  several  steps.  Automatic  punching  from  BI  and  task-rating 
mark-sense  cards  to  punch  cards  was  done  on  the  reproducer  (IBM  519).  Rating  msirks  labeled 
A  through  E  on  mark-sense  rating  cards,  but  regarded  as  quantitie**  1  to  3  fot  analysis,  were 
punched,  widiout  conversion,  according  to  their  position  on  the  mark-sense  card  column,  regard¬ 
less  of  the  digits  normally  represented  by  these  positions.  Punching  for  the  two  ratings  was 
in  separate  punch  card  columns.  Punches  were  made  in  both  rating  columns  for  the  tasks  marked 
as  done  (in  the  case  of  the  covered  and  uncovered  rating  treatments)  and  for  the  tasks  marked 
not  done  (in  the  case  of  the  single-step  treatment). 

The  deck  of  punch  cards  thus  produced  was  edited  by  use  of  the  statistical  sorter  (IBM 
101).  BI  punch  cards  with  missing  information  were  sorted  out.  Task-rating  cards  sorted  out 
were  those  with  one  or  more  instances  of  incompatibility  between  rating  punches  or  lack  of 
punches  on  the  one  hand  and  punches  to  indicate  tasks  done  or  not  done  on  the  other  hand. 

These  rejected  punch  cards  were  then  matched  on  the  collator  (IBM  089)  with  their  correspond¬ 
ing  mark-sense  source  cards.  Errors  were  hand-ccrrccted  on  BI  and  task-rating  mark-sense 
cards,  and  corrected  punch  cards  were  made  on  the  reproducer  or  key  punch  (IBM  024). 

The  corrected  punch  card  deck  was  reproduced  with  task-done  and  task-not-done  punches 
omitted  and  with  a  zero  punch  added  for  each  of  the  two  rating  columns  having  no  rating  punches. 
A  second  editing  sorted  out  cards  with  incompatibilities  between  zero  and  rating  punches  in  the 
same  column  and  between  zero  punches  of  the  two  columns  for  the  same  task;  corrections  were 
made  as  before. 

The  sorter  (IBM  083)  <uid  interpreter  (IBM  337)  were  used  during  the  machine  processing 
as  needed  to  re-sort  and  to  replace  interprered  ikto  on  corrected  punch  cards. 

In  order  to  have  a  deck  to  compare  with  the  deck  automatically  produced  from  mark-sense 
cards,  operators  punched  the  same  data  by  hand  on  the  key  punch.  They  read  die  responses 
from  the  mark-sense  source  cards  as  if  such  dau  had  been  recorded  on  inventory  booklet  pages. 
Ratings  labeled  A  to  E  were  punched  as  1  to  3.  In  the  case  of  responses  written  and  also  mark- 
sensed,  die  written  responses  were  followed  except  in  the  case  of  obvious  errors  such  as 


5 


out-of-range  ratings.  Routine  checking  was  done  on  the  verifier  (IBM  026).  The  resulting  deck 
of  key  punch  cards  and  die  deck  of  punch  cards  automatically  produced  from  mark-sense  cards 
were  to  be  compared  by  the  collator.  Before  die  comparison  was  made,  however,  the  rating 
cards  of  die  mark-sense  punch  deck  were  converted  on  the  computer  (IBM  630).  The  punches 
for  1  to  3  and  zero  were  made  in  die  column  positions  normally  representing  these  digits,  as 
on  the  key  punch  deck. 

In  order  to  compare  the  converted  mark-sense  punch  deck  with  the  key-punch  deck,  a 
machine  roster  was  made,  on  the  accounting  machine  (IBM  407),  of  each  pair  of  cards  of  die 
two  decks  having  one  or  more  differences  in  punches;  and  die  differences  were  visually  iden¬ 
tified  and  labeled.  The  erring  version,  mark-sense  or  key  punch,  was  ascertained  by  visually 
comparing  each  discrepancy  on  the  roster  with  die  original  datum  on  the  mark-sense  source 
card. 


Analysis.  The  procedure  for  analyzing  the  differences  in  accuracy  of  punching  between 
die  mark-sense  and  key-punch  mediods  included  grouping  incumbents  by  administrative  treatment 
and  skill  level.  This  grouping  was  also  used  for  the  analysis  of  differences  in  the  effect  of 
die  three  treatments.  Because  of  small  numbers,  incumbents  in  skill  level  9  were  combined 
with  diose  in  level  7,  those  in  level  1  were  combined  with  those  in  level  3,  and  incumbents  in 
the  Correction  Ladder  (77XX1)  were  combined  with  those  in  the  conventional  Air  Police  Ladder 
(77XX0).  Thus  nine  groups  were  formed  according  to  treatment  (coveted  rating,  uncovered 
rating,  and  single-step  rating)  and  skill  level  (9*7,  3,  and  3*1).  For  the  comparison  of  the 
mark-sense  and  key-punch  methods  in  terms  of  types  of  rating  punch  errors,  and  in  terms  of 

time  and  dollar  economy  of  processing,  die  grouping  was  by  treatment  only. 

« 

In  the  analysis  of  punching  accuracy,  BI  and  identification  errors  were  compared  by  num¬ 
ber  alone;  but  for  errors  of  rating,  both  number  and  percentage  of  total  possible  errors  (in  terms 
of  cases,  cards,  and  punches)  were  used.  The  possible  number  of  rating  punch  errors  for  com¬ 
puting  percentages  was  taken  to  be  the  total  number  of  punches  where  tasks  had  been  actually 
rated  on  both  factors;  zero  punches  were  excluded  as  representing  omissions,  not  ratings.  In¬ 
clusion  of  zero  punches  would  have  resulted  in  a  large  spurious  deflation  of  percentages.  In 
counting  errors,  however,  an  omission  punched  as  a  rating,  as  well  as  a  rating  punched  as  an 
omission,  were  included  as  punching  errors.  The  punching  of  multiple  responses  to  the  same 
factor  or  of  responses  where  only  one  factor  was  rated  as  anything  except  zero  was  also  counted 
as  an  error.  In  computing  die  percentages  of  cards  with  one  or  more  radng  punch  errors,  the 
possible  number  of  cards  was  uken  to  be  the  number  having  one  or  more  tasks  rated  on  both 
factors. 


Further  analysis  of  rating  punch  accuracy  consisted  of  a  comparison  of  statistics  computed 
from  data  produced  by  the  mark-sense  and  key-punch  methods  with  statistics  computed  from  cor- 
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the  two  ratings,  and  correlation  between  the  ratings. 


in  die  comparison  of  types  of  rating  punch  errors  between  the  mark-sense  and  key-punch 
methods,  only  number  of  errors  was  used.  Likewise,  niunber  of  hours  and  dollars  was  used  in 
the  comparison  of  time  and  cost  economy. 

The  comparison  of  ue  three  treatments  to  determine  differential  effect  of  the  first  rating 
on  the  second  was  made  in  terms  of  correlation  between  the  two  ratings,  computed  from  the 
conected  data. 
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3.  RESULTS 


Survey  Adniniatration 

In  the  administration  of  the  survey,  smoodiness  of  procedure  coofonoed  to  accepted 
standards.  The  overall  time  for  administration  ranged  from  75  to  90  minutes  for  the  covered 
and  uncovered  rating  treatments  and  hcom  60  to  75  minutes  ftx  the  single-step  treatment.  In 
each  treatment,  the  BI  card  took  about  14  minutes.  On  the  basis  of  die  time  required  fm  this 
146-task  inventory,  it  is  estimated  diat  an  inventory  with  300  task  statements  of  the  same  type 
could  be  administered  in  about  two  and  one-half  hours  for  the  covered  and  uncovered  rating 
treatments,  and  in  about  two  hours  for  the  single-step  treatment. 

Processing  Accuracy 

BI  Card  Errors,  Only  6  errors  were  found  in  the  367  mark-sense  BI  punch  cards.  Three 
coded  job  assignment  titles  were  incoaectly  punched  altfaou^  correctly  recorded  on  marit-sense 
cards.  The  other  three  errors  were  attributed  to  incorrect  mark-sensing  which  had  escaped  the 
scrutiny  of  the  scanners.  No  errors  were  found  in  die  key  punch  BI  cards. 

Task-Rating  Card  Errors.  In  the  5505  cards  of  the  mark-sense  task-rating  punch  deck, 
the  code  niunber  for  machins-room  identification  of  the  study  was  omitted  from  7  cards  which 
had  been  repunched.  This  code  number  was  originally  gang-punched.  In  the  key-punch  task¬ 
rating  deck,  the  duty  letter  was  incorrectly  punched  on  diree  cards. 

Of  the  5505  pairs  of  task-rating  cards  produced  by  two  processing  methods,  2857 
contained  no  entries  because  tasks  were  not  performed  by  the  raters.  Of  the  2648  pairs  of 
cards  with  at  least  one  task  rated,  238  pairs  had  one  or  more  rating  punch  differences,  for  a 
total  of  477  discrepancies  in  rating  punches  between  die  mark-sense  and  key-punch  versions, 
as  shown  in  Table  2.  The  data  given  are  the  number  and  percentage  of  task-rating  errors  for 
each  method,  by  treatment  and  skill  level.  No  definite  trends  related  to  skill  levels  appeared 
in  the  table.  Errors  were  computed  in  terms  of  cards  and  cases  as  well  as  punches  to  ascertain 
whether  there  was  a  tendency  for  errors  to  be  concentrated  on  a  few  cards  or  cases.  Since  no 
such  concentration  appeared,  furdier  ‘consideration  of  errors  will  be  in  terms  of  punches  only. 

The  standard  error  formula  used  to  compute  the  significance  of  the  difference  between  the 
percentages  of  errors  for  die  mark-sense  and  key-]Mnch  methods  is  one  suitable  for  correlated 
samples  with  small  Ns  and  small  percentages  (McNemar,  1949,  P*  80):  die  percentage  of  the 
two  samples  combined  is  used  in  place  of  the  separate  percentages  appearing  in  the  conventional 
formula.  As  may  be  seen  in  Table  2,  die  key-punch  method  was  consistently  more  accurate  dian 
the  mark-sense  method.  Of  the  48  comparisons  made,  7  had  Ns  and  percentages  too  souili  to 
justify  use  of  the  t  technique  (McNemar,  1949,  p.  80).  All  of  die  totals  for  treatments  and  skill 
levels  ske^  differesces  sigsiflcsst  beyosd  die  *001  levels 

The  mark-sense  method  had  punching  error  percentages,  for  all  three  treatments,  in  excess 
of  the  1  percent  considered  as  the  maximum  tolerable  for  experimental  test  scoring.  The  key¬ 
punch  method,  on  the  other  hand,  had  only  negligible  percentages  for  the  treacaests  and  a  grand 
toul  of  only  .1  percent. 

Table  3  shows  die  frequency  of  difforent  types  of  errors  made  in  rating  punches.  It  is 
apparent  that  the  number  of  mark-sense  receding  errors  uncorrected  visual  scanning  and 
machine  editing  was  negligible  for  all  treatments.  The  type  of  error  occurring  most  jErequentiy 
in  the  covered  and  uncovered  rating  treatments  of  the  mark-sense  method  was  ''omission,  but 
1  to  5  punched";  whereas  the  most  frequently  occurring  ^pe  in  the  single-step  treatment  was 
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Table  2.  Diatrlbution  of  Rating  Punch  Enrors  Made  in 
Mark-Sense  (MS)  and  Key-Punch  (KP)  Methods 
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"1  to  3  marked,  but  zero  punched.**  Since  this  last  type  accounted  for  84  of  die  97  errors  in 
the  single-step  treatment  of  the  mark-sense  method,  and  since  this  kind  of  error  is  due  to 
machine  rather  than  to  operator  failure,  the  mark-sense  cards  were  rerun  through  a  reproducer 
after  an  IBM  engineer  had  made  a  maintenance  check.  In  this  rerun  only  13  punching  errors 
were  made:  4  "wrong  digit  (1  to  5)  punched**;  3  "omission,  but  1  to  5  punched";  and  8  "1  to 
3  marked,  but  zero  punched.**  Eight  of  die  errors  repeated  errors  made  in  the  original  run 
because  of  uncorrected  mark-sense  marks;  the  other  seren  were  new  errors.  The  rate  of 
punching  errors  for  this  mark-sense  rerun,  15  of  a  possible  7960,  was  only  .2  percent.  These 
rerun  punch  cards  were  not  used  for  any  of  the  tabulations  of  this  study. 


Table  3.  Distribatioa  of  Rating  Pnack  Errors  by  of  Errors 


TYPE  OF  ERROR 

MARK-SENSE  METHOD 

KEY-PUNCH  METHOD 

COV 

UNCOV 

l-STEP 

TOT 

COV 

UNCOV 

l-STEP 

TOT 

Wrong  digit  (1  to  5)  punched 

52* 

33 

3 

88 

0 

2 

0 

2 

Omission,  but  1  to  5  punched 

72 

118* 

2 

192 

1 

3 

0 

4 

1  to  5  marked,  but  zero  punched 

59 

20 

84 

163 

6 

2 

10 

Uncorrected  mark-sense  error 

3 

7 

8 

18 

- 

- 

- 

— 

Total 

186 

178 

97 

461 

3 

11 

2 

16 

*Oae  out-of-range  punch  (6-9)  included. 


Table  4  presents,  by  treatment  and  skill  level,  the  number  of  tasks  rated  and  summary 
statistics  computed  from  data  of  die  mark-sense  punch-card  deck,  data  of  the  key-punch  deck, 
and  corrected  data.  It  shows  means  and  standard  deviations  of  the  ratings  for  time  spent 
(TS)  and  training  and  experience  (T&E)  and  conelations  between  the  ratings.  Two  definite 
trends  related  to  skill  levels  appear  in  the  table,  both  of  which  have  been  observed  in  other 
studies  (Gragg,  1962).  ’  Those  of  higher  levels  rate  more  task?  and  rate  training  and  exper¬ 
ience  higher.  In  spite  of  substantial  discrepancies  in  the  number  of  tasks  shown  as  rated, 
especially  by  the  d&u  of  the  mark-sense  method,  the  differences  between  the  summary' 
statistics  horn  corrected  data  are  small.  The  largest  difference  for  a  mean  is  .05;  for  a 
standard  deviation,  .02;  and  for  a  correlation,  .01. 


rtf 


To  compare  the  cost,  in  hours  and  in  dollars,  of  the  method  of  key  punching  from 
inventory  booklets  with  that  of  the  mark-sense  method,  it  was  necessary  to  estimate  die 
time  dial  would  have  been  expended  in  key  punching  the  source  data  of  the  present  study 


’  And  unpublished  surveys  carried  out  by  the  Personnel  Research  Laboratory  (PR6) 
under  the  direction  of  Dr.  J.  E.  Morsh. 
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Table  4.  Statistics  of  Tiiae  Spest  (TS)  sad  Training  and  Experience  (T*E)  Ratings  Coaiputed  Fro« 
Mark-Sense  (MS),  KeyPanch  (KP),  and  Corrected  (Cor)  DaU 
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from  responses  recorded  In  ioTentocy  booklets  iostesd  of  on  msrk-seose  cards.  For  d»s  esti¬ 
mate,  time  actually  spent  on  key  punching  and  verifying  similar  inventory  booklet  data  of 
previous  studies  was  used  (Gragg,  1962).?  For  the  mark-sense  processing  time,  die  hours 
reported  for  this  study  by  the  key-punch  supervisor  were  used.  Time  spent  on  programming 
and  on  wiring  control  panels  for  machine  operation  was  not  included  because  it  would  be 
expended  only  on  a  one-time  basis  during  planning  and  setting  up  a  mark-sense  processing 
system.  The  time  in  hours,  by  treatment,  is  given  for  the  two  processing  mediods  in  Table  5. 


Table  5.  Tiaie  Required  for  Paaehiig  Capds  by  the  Mark-Sease  Method 
and  tbe  Key-PauciHMetbod 


MARK-SENSE  METHOD 

KEY.PUNCH  METHOD 

TREATMENT 

MAN  HOURS 
(SCANNING) 

MAN4IACHINE 

HOURS 

TOTAL 

HOURS 

MAN4IACHINE 

HOURS* 

Covered 

19.40 

11.57 

30.97 

16.52 

Uncovered 

20.62 

11.52 

32.14 

17.50 

Single  Step 

4.95 

7.35 

12.30 

17.50 

Total 

44.97 

30.44 

75.41 

51.52 

*  Estiaiated  from  time  requited  for  pnaebiag  siaular  data  from  iaveatoty 
booklets. 


According  to  Table  3,  the  marie-sense  cards  of  the  treatments  (covered  and  uncovered) 
requiring  two  rating  steps  took  much  more  time  to  process  than  did  those  of  the  single-step 
treatment.  The  difference  was  due  to'  die  format  of  the  cards.  The  separate-step  cards 
called  for  blank  columns  for  tasks  not  performed  and  columns  with  three  marks  for  tasks 
performed,  one  mark  for  performance  and  two  for  ratings.  The  single-step  card  columns  provided 
for  two  rating  marks  for  tasks  done  and  one  mark  for  tasks  not  done.  The  time  spent  in  the  de¬ 
velopment  of  i.  processing  system  for  the  two-step  card  was  included  in  die  hours  reported  by 
machine  operators  for  the  processing  itself.  No  developmental  time  was  required  for  die  single- 
step  card  system,  already  in  use  for  punching  test  item  cards.  Accordingly  the  time  for  proc¬ 
essing  the  single-step  cards  was  regarded  es  repreeesteti'se  of  the  operatiotial  aark-sease 
processing,  and  was  used  in  comparing  the  mark-sense  with  the  key-punch  method.  As  shown 
in  Table  $,  tbe  total  time  for  processing  the  125  cases  of  the  single-step  treatment  was  17.3 
hours  for  the  key-punch  method  and  12.3  hours  for  the  mark-sense  method,  about  30  percent 
less. 


^And  an  uapubliehed  study  carried  out  by  Persoaael  Research  Laboratory  (PRB)  under 
the  direction  of  Lt  Col  J.L^  Madden. 
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Estimates  of  die  dollar  costs  of  die  mark-sense  and  key-punch  methods  of  punching 
the  single-step  cards  are  given  in  Tables  6  and  7.  Machine-hour  cost  was  computed  from 
the  monthly  charge  on  the  IBM  basis  of  176  hours  per  month.  Man-hour  costs  were  computed 
according  to  local  wage  rates,  widi  no  allowance  for  supervisor  pay.  In  computing  card 
cost,  the  standard  price  per  thousand  was  used,  and  no  allowance  was  made  for  wastage. 
The  cost  of  processing  the  125  cases  of  the  single-step  treatment  was  |32.06  for  the  key¬ 
punch  mediod  and  142.44  for  the  mark-sense,  about  32  percent  mote. 


Table  6.  Gist  of  Paacking  Siagle-Step  Cards  by  Mark-Sease  Method 


COST 

MACHINE  COSTS 

NO.  HRS 

PER  HR 

COST 

Sorter  (083) 

0.83 

10.76 

1  0.63 

CoUator  (089) 

1.50 

1.25 

1.88 

Stat  Sorter  (101) 

1.72 

2.84 

4.88 

Reproducer (519) 

3.13 

1.63 

5.10 

Interpreter  (557) 

0.17 

1.04 

0.18 

Machine  Total 

112.67 

OPERATOR  COSTS 

Machine  Operation 

7.35 

2.35 

17.27 

Scanning 

4.95 

1.50 

7.42 

Operator  Total 

COST 

124.69 

MATERIALS 

NO. 

PERM 

Punch  Cards 

4000 

1.27 

t  5.08 

TOTAL  142.44 


Table  7.  Cost  of  Paackiag  Siagle-Step  Cards  by  Key-Paack  Method 


MACHINE  COSTS 

NO.  HRS 

COST 

PER  HR 

COST 

Key  Punch  (024) 

9.23 

10.24 

12.22 

Verifier  (026) 

8.27 

0.30 

2.48 

Machine  Total 

1  4.70 

OPERATOR  COST 

Machine  Operation 

17.50 

1.50 

126.25 

MATERIALS 

Punch  Cards 

NO. 

875 

COST 

PERM 

1.27 

1  1.11 

132.06 


12 


TOTAL 


Effect  of  AdHiaistnitive  IVeatHeats 

As  shown  in  Table  4  under  the  corrected  data  columns,  the  correlations  of  the  time* 
spent  ratings  with  the  training-and-ezperience  ratings,  for  all  skill  lerels  combined,  are  .31 
for  the  covered  rating,  .41  for  die  uncovered  rating,  and  .30  for  the  single-step  treatment. 

The  differences  (P<.01,  as  computed  by  the  z-transformation  technique)  are  appreciable  in 
size  and  are  in  the  direction  expected. 

The  number  of  listed  tasks  rated  and  the  number  of  write-in  task  statements  added  were 
not  hypothesized  to  be  affected  by  differences  in  administrative  treatments.  The  average 
numbers  of  usks  rated,  by  each  case,  based  on  the  corrected  data  columns  of  Table  4,  show 
only  negligible  differences  between  the  three  treatments.  In  the  production  of  write-in  state¬ 
ments,  however,  not  shown  in  the  table,  each  of  the  covered  and  uncovered  rating  treatments 
resulted  in  118  statements  for  all  levels  combined  while  in  the  single-step  treatment  only 
37  statements  were  obtained. 


4.  DISCUS5I(m 


Survey  Administration 

The  administrative  procedures  adapted  from  the  booklet  recording  technique  to  die  mark- 
sense  recording  method  proved  to  be  satisfactory.  The  overall  administrative  time  estimates 
reported  make  no  allowance  for  greater  speed,  due  to  familiarity  with  mark-sensing,  that  could 
be  expected  on  the  latter  items  of  a  longer  inventory.  It  was  apparent  also  that  time  could  be 
saved  by  shortening  the  administrative  directions.  It  seems  likely,  therefore,  that  an  Inventory 
of  300  comparable  task  statements  could  be  administered  by  any  one  of  the  treatments  within 
a  time  limit  of  about  two  hours.  This  estimated  time  is  comparable  to  that  required  for  similar 
inventories  currently  used  with  responses  recorded  in  booklets. 

Processing  Accnracy 

The  key-punch  method  was  found  to  be  highly  accurate  for  both  BI  and  task-rating  cards, 
as  may  be  expected  when  all  data  are  checked  on  the  verifying  machine  (IBM  026).  For  the 
mark-sense  method,  die  punching  accuracy  for  data  on  BI  cards  and  for  the  identification  data 
on  task-rating  cards  was  also  satisfactory.  The  processing  of  rating  responses  on  die  mark- 
sense  cards,  however,  was  not  so  accurate  as  desired  since  the  errors  made  in  punching  ratings 
in  all  three  treatments  were  in  excess  of  die  1  percent  generally  considered  as  the  makimum 
tolerable.  In  spite  of  these  inaccuracies,  the  summary  statistics  computed  from  die  mark-sense 
data  showed  only  negligible  deviations  from  the  correct  statistics.  It  might  be  concluded  diat 
the  mark-sense  method  is  quite  adequate  for  purposes  of  occupational  analysis  researdi.  This 
conclusion  would  hold,  however,  widi  an  error  rate  greater  than  1  percent,  only  so  long  as  errors 
AM  £ts  chsy  sppSLTsntly  is  this  study*  Siscc  errors  nii^t  veil  occur  systeinftti* 

cally,  the  error  rate  must  be  kept  below  1  percent  if  the  conclusion  is  to  hold.  The  rating- 
punch  error  rate  (.2  percent)  of  the  rerun  of  the  single-step  cards  indicates  that  hi^ly  accurate 
punching  can  be  obtained  with  a  reproducer. .  Unless  the  reproducer  is  maintained  in  top  condi¬ 
tion,  however,  mark-sense  processing  cannot  be  depended  upon  without  additional  verification. 

Economy  of  Processing 

The  mark-sense  method  showed  30  percent  fewer  mao  and  machine  hours  than  the  key¬ 
punch  method.  Since,  however,  the  typical  data  p’oeessing  organization  has  several  key  punch 
and  verifier  machines  for  every  reproducer  or  sutistical  sorter,  die  apparent  time  advantage 
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of  the  merk'sense  method  is  actually  not  meaningful.  In  dollar  economy,  also,  the  mark-sense 
method  failed  to  show  an  advantage.  Because  wages  paid  to  machine  room  technicians  are 
hi^er  than  those  paid  to  key  punch  operators  and  clerical  personnel,  the  dollar  cost  of  the 
Buurk-scnse  method  turned  out  to  be  32  percent  more  than  that  of  the  key  -punch  method. 

Effect  of  Anuaiaiatrative  IVeatmeats 

Having  the  first  rating  covered  while  making  the  second  resulted  in  the  least  correlation 
between  factor  ratings;  the  single-step  treatment  procedure  of  making  bodi  ratings  of  each  task 
before  proceeding  to  the  next  task  produced  the  largest  correlation;  and  the  currently  used 
technique  of  making  ratings  in  separate  steps  but  with  the  first  in  view  during  the  recording 
of  the  second  showed  correlation  intermediate  in  size. 

In  the  p-  oduction  of  write-in  task  statements,  both  of  the  separate-step  treatments  were 
superior  to  the  single-step  treatment,  apparently  because  the  incumbents  were  required  to 
indicate  whether  or  not  every  listed  task  was  done  before  adding  tasks  that  were  not  listed. 
Asking  incumbents  merely  to  read  the  listed  tasks  before  adding  task  statements  [produced  less 
than  half  as  many  write-ins.  The  difference  between  the  separate-step  treatments  and  the 
single-step  treatment  in  teems  of  the  number  of  listed  tasks  rated,  ho'.~^7£r,  proved  to  be 
negligible. 


5.  CONCLUSIONS 

Correlation  between  rating  factors  is  reduced  by  having  previous  ratings  out  of  sight 
while  later  ratings  are  mnde. 

Requiring  incumbents  to  respond  to  listed  tasks  beforehand  increases  the  production  of 
write-in  task  statements. 

The  mark-sense  method  of  producing  punch  cards  offers  so<n:>  man-hour  economy.  Tlie 
period  of  time  required  for  producing  a  given  amount  of  punch  cards,  however,  will  normally 
be  less  for  the  key-punch  method  because  of  the  number  of  key  punch  and  verifier  machines 
in  a  typical  data  processing  organization. 

In  dollar  cost,  the  mark-sense  mediod  is  more  expensive  than  the  keypunch  method 
because  machine  technicians  are  paid  more  than  key  punch  operators. 

The  key-punch  method,  including  verification  of  all  punching,  is  more  accurate  chan  the 
Buuk-sense  method.  The  mark-sense  method,  however,  is  satiifiictorily  acciuate  when  the 
reproducer  is  properly  maintained. 

The  mark-sense  method  of  producing  punch  cards,  while  feasible,  has  no  apparent  advan¬ 
tage  over  the  key-punch  mediod  in  the  collection  of  occupational  data. 

kfarkiog  nonperformance  rather  than  performance  of  tasks  may  reduce  the  time  required 
in  working  out  a  mark-sense  processing  technique,  but  once  the  technique  is  developed  should 
mxkt  no  difference  in  either  accuracy  or  economy. 
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